{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# adaboost 回归算法\n",
    "参考[https://blog.csdn.net/weixin_43597208/article/details/145905111]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 算法概念\n",
    "Adaboost 是 Adaptive Boosting 的缩写，是一种集成机器学习算法，可用于各种分类和回归人物。他是一种监督学习算法，用于通过将多个弱学习器或基础学习算法(例如决策树)组合成一个强学习器来对数据进行分类和回归人物。AdaBoost的工作原理是根据先前分类的准确性对训练数据集中的实例进行加权，也就是说 Adaboost构建了一个模型，并为所有数据点分配了相同的权重，然后，它将更大的权重应用与错误分类的点。在模型中，所有权重较大的点都会被赋予更大的权重。它将继续训练模型，知道返回较小的错误，所以,\n",
    "* Boosting 是一个迭代的训练过程。\n",
    "* 后续模型更加关注前一个模型中错误的样本\n",
    "* 最终预测是所有预测的加权组合\n",
    "  \n",
    "如下图所示:\n",
    "![image.png](https://i-blog.csdnimg.cn/direct/6c1cb4c0957446dc8161679b4c96080e.png#pic_center)\n",
    "弱学习器：AdaBoost从弱分类器开始，弱分类器是一种相对简单的机器学习模型，其准确率仅略高于随机猜测率。弱分类器通常使用一个浅层决策树。\n",
    "权重初始化：训练集中的每个实例最初被分配一个相等的权重。这些权重用于在训练期间赋予更困难的实例更多的重要性。\n",
    "迭代（Boosting）：AdaBoost执行迭代来训练一组弱分类器。在每次迭代中，模型都会尝试纠正组合模型到目前为止所犯的错误。在每次迭代中，模型都会为前几次迭代中被错误分类的实例分配更高的权重。\n",
    "分类器权重计算：训练较弱的分类器的权重根据其产生的加权误差计算。误差较大的分类器获得较低的权重。\n",
    "更新实例权重：错误分类的示例获得更高的权重，而正确分类的示例获得更低的权重。这导致模型在后续迭代中专注于更困难的示例。\n",
    "加权组合弱分类器：AdaBoost将加权弱分类器组合生成一个强分类器。这个过程的关键在于，每个弱分类器的权重取决于它在训练过程中的表现，表现好的分类器会得到更高的权重，而表现不佳的分类器会得到较低的权重。\n",
    "最终输出：最终输出是所有弱分类器组合而成的强分类器，该最终模型比单个弱分类器准确率更高。\n",
    "  AdaBoost 的重点在于，通过这个迭代过程，模型能够聚焦于难以分类的样本，从而提升系统的整体性能。AdaBoost 具有很强的鲁棒性，能够适应各种弱模型，是一种强大的集成学习工具。算法流程如下所示：\n",
    "\n",
    "![image.png](https://i-blog.csdnimg.cn/direct/9e10b71196a6453780cbbfbef29f5cf9.jpeg#pic_center)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 算法原理\n",
    "\n",
    "## (一) 分类算法基本思路\n",
    "\n",
    "### **1.训练集合和权重初始化**\n",
    "训练集: \n",
    "$$T = {(x_1,y_1),(x_2,y2),\\cdot,(x_m,y_m)},x_i \\in R^n$$\n",
    "\n",
    "AdaBoost 算法从样本权重初始化开始，每个样本被赋予相等的权重。\n",
    "$$ D(t) = (w_t1,w_t2,\\cdots,w_tm); w_{1,i} = frac{1}{m}, i = 1,2,\\cdots,m $$\n",
    "其中，所有的权重$w_{1,i} 的总和为 1，表示每个样本的初始重要性是相等的。\n",
    "\n",
    "### **2.弱分类器的加权误差** \n",
    "\n",
    "这里假设我们是二元分类问题，在第$t$轮的迭代中，输出为${-1,1}$，则$t$个弱分类器$G_t(x)$在训练集上的加权误差率为：\n",
    "$$ \\epsilon_t = P(G_t(x_i) \\neq y_i) = \\sum_{i=1}^{m}w_{ti}I(G_t(x_i) \\neq y_i)$$\n",
    "\n",
    "* $\\epsilon_t$表示错误率的变量，下标$t$表示第$t$轮迭代。\n",
    "* $P(G_t(x_i) \\neq y_i)$:\n",
    "    * $G_t 是第$t$轮 预测模型，他是一个函数能对输入$x_i$进行预测。\n",
    "    * $x_i$ 是训练集中的第$i$个样本。\n",
    "    * $y_i$ 是训练集中第$i$个样本的标签。\n",
    "    * $P(G_t(x_i) \\neq y_i)$ 表示第$t$轮，模型$G_t$对第$i$个样本的预测与真实标签$y_i$不一致的概率。\n",
    "* $\\sum_{i=1}^{m}{w_{ti}}I(G_t(x_i) \\neq y_i)$:\n",
    "    * $m$表示训练样本总和\n",
    "    * $w_{ti}$是第$t$轮第$i$个样本的权重。在一些算法中，不同样本的重要性不同，权重可以调整对样本整体结果的贡献程度。\n",
    "    * $I(\\cdot)$是指函数，当括号内的条件成立是，$I$取1，否则取0。在这里是$G_t(x_i) \\neq y_i$,即预测错误，$I(G_t(x_i) \\neq y_i) = 1$。如果$G_t(x_i) = y_i$,则$I(G_t(x_i) \\neq y_i) = 0$。\n",
    "* 整个求和就是对每个样本的权重和指示函数值的乘积进行求和，以此计算出在第$t$轮时模型的预测错误率。\n",
    "\n",
    "\n",
    "这里$\\epsilon_t$是第$t$个弱分类器的加权误差率。$w_{ti}$是样本$i$在第$t$轮的权重，反映了在这一轮中该样本的重要性。加权误差表示的是分类错误的样本的权重和，如果$\\epsilon_t$接近 0，说明分类起表现良好，错误率很低；如果$\\epsilon_t$接近 0.5，说明分类器的表现几乎是随机猜测。\n",
    "\n",
    "### **3.弱分类器权重**\n",
    "\n",
    "对于每一轮的弱分类器，权重系数的表达公式为:\n",
    "$$ \\alpha_t = \\frac{1}{2}log(\\frac{1-\\epsilon_t}{\\epsilon_t})$$\n",
    "\n",
    "这个公式表明了若分类器的表现与其权重之间的关系:\n",
    "<br>\n",
    "如果$\\epsilon_t$很小，则表示弱分类器表现好，那么$\\alpha_t$会很大，表示这个分类器在最终组合中有较大的权重。$\\epsilon_t$越接近 0.5，表示分类器效果接近随机猜测，那么$\\alpha_t$越接近 0，表示该分类器在最终组合中权重很小，如果$\\epsilon_t$大于 0.5,理论上这个分类器效果是反向的，意味着他的分类错误率超过50%。因此 Adaboost 会停止训练。\n",
    "\n",
    "问题思考:\n",
    "* 为什么这儿是$\\frac{1}{2}$ 避免权重幅度过大或者过小。\n",
    "\n",
    "### **4.Adaboost分类损失函数**\n",
    "\n",
    "Adaboost 是一种基于加法模型和前向分布算法的分类算法，通过一系列弱分类器的组合来构建一个强分类器，核心思想是根据分类错误率动态调整样本的权重，使得分类器能更好地处理难以分类的样本。\n",
    "在 Adaboost 中，最终的强分类器:$H_t(x)$,是若干个弱分类器的加权组合。\n",
    "$$ H_t(x)=sign(\\sum_{t=1}^{T} {\\alpha_t}{G_t(x)})$$\n",
    "\n",
    "* $\\alpha_t$ 是第$t$轮弱分类器的权重系数。\n",
    "* $G_t(x)$ 是第$t$轮弱分类器对输入$x$的预测输出。\n",
    "\n",
    "通过向前分布学习步骤，强分类器逐步构建为:\n",
    "$$H_t(x)=H_{t-1}(x)+{\\alpha_t}{G_t(x)}$$\n",
    "\n",
    "Adaboost的损失函数定义为指数损失函数，其公式为：\n",
    "$$ argmin_{G_t} \\sum_{i=1}^{m}exp(-y_iG_t(x_i))$$\n",
    "\n",
    "其中:\n",
    "* $y_i$ 是训练集中第$i$个样本的标签。取值${-1,1}$。\n",
    "* $h_t(x_i)$是分类器$h_t$ 对输入$x_i$的预测输出。\n",
    "\n",
    "利用前向逐步递推公式$h_t(x_i)= h_{t-1}(x_i)+\\alpha_t h_t(x_i)$，损失函数可以写成:\n",
    "$$ (\\alpha_t,G_t(x)) = arg \\min_{\\alpha,G} \\sum_{i=1}^{m}exp[{-y_i}{(h_{t-1}(x_i)+{\\alpha}{G(x_i)})}]$$\n",
    "\n",
    "定义样本权重$w_{t,i}^{'}$ 为:\n",
    " $$ w_{t,i}^{'} = exp[-y_i(h_{t-1}(x_i))]$$\n",
    "\n",
    "它的值不依赖于$\\alpha_t 和 G_t(x)$,只与$h_{t-1}(x_i)$有关。\n",
    "因此损失函数可以写成:\n",
    "$$ (\\alpha_t,G_t(x)) = arg \\min_{\\alpha,G} \\sum_{i=1}^{m}{w_{t,i}^{'}}exp[{-y_i}{\\alpha}{G(x_i)}]$$\n",
    "\n",
    "为了找到最优的弱分类器$G_t(x)$,可以将损失函数展开:\n",
    "$$\\sum_{i=1}^{m}{w_{t,i}^{'}}exp[{-y_i}{\\alpha}{G(x_i)}] =  \\sum_{y_i=G_t(x_i)}{w_{t,i}}e^{-\\alpha}+\\sum_{y_i\\neq G_t(x_i)}{w_{t,i}}e^{\\alpha}$$\n",
    "\n",
    "因此可得到最优分类器$G_t(x)$ 选择为:\n",
    "$$ G_t(x) = arg \\min_{G} \\sum_{i=1}^{m}{w_{t,i}^{'}}I(y_i \\neq G(x_i))$$\n",
    "将$G_t(x)$ 代入损失函数中,对$\\alpha$求导并令导数为 0,可得:\n",
    "$$\\alpha_t = \\frac{1}{2}log(\\frac{1-\\epsilon_t}{\\epsilon_t})$$\n",
    "\n",
    "其中$\\epsilon_t$ 是第$t$轮弱分类器的加权误差率。\n",
    "$$ \\epsilon_t = \\frac{\\sum_{i=1}{m}{w_{t,i}^{'}}{I(y_i \\neq G_t(x_i))}}{\\sum_{i=1}^{m} {w_{t,i}^{'}}} = \\sum_{i=1}^{m}w_{t,i}I({y_i \\neq G_t(x_i)})$$\n",
    "\n",
    "在第t+1轮中，样本权重会根据弱分类器的表现进行更新。对于分类错误的样本，其权重会增大，从而在下一轮中对这些样本给予更多的关注。通过以上推导，可以得到Adaboost的弱分类器样本权重更新公式。\n",
    "\n",
    "### 5.样本权重更新\n",
    "  接下来，计算AdaBoost 更新样本的权重，以便在下一轮训练中更加关注那些被当前弱分类器错误分类的样本。利用 $h_t(x) = h_t(x-1)+\\alpha_t G_t(x)$ 和 $w_{t,i}^{'}=exp[-y_i(h_{t-1}(x_i))]$，可以得到：\n",
    "$$ w_{t+1,i} = \\frac{w_{t,i}}{Z_T} exp[-\\alpha_t y_i G_t(x_i)]$$\n",
    "\n",
    "其中 ，\n",
    "* $\\alpha_t$ 是第$t$轮弱分类器的权重，\n",
    "* $y_i$ 是样本的真实标签\n",
    "* $G_t(x_i)$ 是第$t$个弱分类器对样本$x_i$的预测结果。\n",
    "这个公式的作用是通过调整权重来强化难以分类的样本.\n",
    "* 如果分类器$G_t(x_i)$ 对样本$x_i$分类错误，即$y_i G_t(x_i)<0 $,会导致$w_{t+1,i}增大，表示这个样本在下一轮中会被赋予更大的权重，模型会更关注它。\n",
    "* 如果分类器$G_t(x_i)$对样本$x_i$分类正确，即$y_i G_t(x_i)>0$，会导致$w_{t+1,i}减小，表示模型会认为这个样本已经很好分类了，下轮可以降低它的重要性。\n",
    "\n",
    "### 6.Adaboost 的强分类器\n",
    "\n",
    "这里 $Z_t$是规范化因子，保证更新后的权重任然是一个概率分布。其中计算公式:\n",
    "$$ Z_t = \\sum_{i=1}{m} w_{t,i} exp(-\\alpha_t y_i G_t(x_i))$$\n",
    "\n",
    "通过这个规范化因子，所有的权重从$w_{t+1,i}被重新调整，使得他们的总和依然为 1，从样本权重更新公式可以看出，分类错误的样本会得到更高的权重，这让下一轮弱分类器更加关注这些难以分类的样本。这种机制逐步强化了对弱分类器表现不好的部分样本的关注，最终通过多次迭代形成一个强分类器:\n",
    "$$ H(x) = sign(\\sum_{t=1}{T} \\alpha_t G_t(x))$$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 回归算法基本思路\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最大误差的计算\n",
    "给定$t$个若学习器$G_t(x)$,其在训练集上的最大误差定义:\n",
    "$$ E_t = max|y_i-G_t(x_i)| i = 1,2,...,m$$\n",
    "\n",
    "计算每个样本上预测值和真实值之间的绝对差值，找到这个差值的最大值。这个最大误差为后续计算每个样本的相对误差提供了一个标准化的尺度，使得每个样本的误差相对该最大误差进行比较。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相对误差计算\n",
    "然后计算每个样本的相对误差\n",
    "$$ e_{ti} = \\frac{|y_i -  G_t(x_i)}{E_t} $$\n",
    "\n",
    "通过相对误差，我们可以统一衡量所有样本的误差，而不受特定样本的绝对误差影响。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 误差损失调整\n",
    "误差损失可以根据不同的度量方式进行调整。\n",
    "\n",
    "如果是线性误差的情况，即直接比较绝对误差:\n",
    "$$ e_ti = \\frac{(y_i-G_t(x_i))^2}{E_t}$$\n",
    "\n",
    "如果使用平方误差，则相对误差为:\n",
    "\n",
    "$$ e_ti = \\frac{(y_i - G_t(x_i))^2}{E_t^2}$$\n",
    "\n",
    "如果我们使用的指数误差，则:\n",
    "$$ e_ti = 1-exp(\\frac{-|y_i-G_t(x_i)|}{E_t}) $$\n",
    "\n",
    "指数误差对较大的误差进行了压缩，使其影响边的非线性。\n",
    "最终得到第$t$个弱学习的误差率：\n",
    "$$ e_t = \\sum_{i=1}^{m} w_{ti} e_{ti} $$\n",
    "\n",
    "反映了第$t$个弱学习器在整个训练集上的整体表现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 权重系数计算\n",
    "\n",
    "对于第$t$个弱学习器，权重系数$\\alpha_t$的计算公式\n",
    "$$ \\alpha_t = \\frac{e_t}{1-e_t} $$\n",
    "\n",
    "这里，权重$\\alpha_t$ 反映了第$t$个弱学习器的重要性。如果误差率$e_t$小，则$\\alpha_t$会较大，表明该弱学习器的重要性较高；反之，误差率大的弱学习器权重较小，这种权重系数分配方法确保了表现更好的弱学习器在组合中获得更大的影响力。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 更新样本权重\n",
    "\n",
    "对于更新样本权重 D，第$t+1$个弱学习器的样本权重集系数为:\n",
    "$$ w_{t+1,i} = \\frac{w_{ti}}{Z_t} {\\alpha_t^1-e_{ti}} $$\n",
    "\n",
    "  样本权重更新的核心思想是，将更多的关注放在那些难以分类的样本上，以便在后续的训练中重点处理这些样本。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 规范化因子\n",
    "\n",
    "这里$Z_t$ 是规范化因子，规范化因子的计算公式:\n",
    "$$ Z_t = \\sum_{i=1}^{m} {w_{ti}} \\alpha_t^{1-e_{ti}} $$\n",
    "\n",
    "  通过这个规范化步骤，保持了样本权重的标准化，使得权重在每一轮迭代中不会无穷增大或减小。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 强学习器\n",
    "  回归问题与分类问题略有不同，最终的强回归器 $f(x)$不是简单的加权和，而是通过选择若干弱学习器中的一个，最终的强回归器为：\n",
    "$$ f(x) = G_{t^\\star} (x) $$ \n",
    "其中, $G_{t^\\star}(x)$ 是所有 $ln(\\frac{1}{\\alpha_t}),t = 1,2,3,...T$ 中位数值对应序号$t^\\star$的弱学习器，这种方法能够在一定程度上避免极端弱学习器的影响，从而更稳定的进行回归预测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 算法的优缺点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 算法的有点\n",
    "* 分类精度高: 作为分类器，Adaboost 可以显著提高分类精度。\n",
    "* 灵活性强：Adaboost 框架下可以使用各种回归或分类模型作为弱学习器，应用广泛。\n",
    "* 简单易理解: 尤其是用于二分类时，算法构造简单且结果容易解释。\n",
    "* 不易过拟合: 相较于其他算法，Adaboost 不容易过拟合。\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 算法的缺点\n",
    "\n",
    "* 对异常样本敏感：在迭代过程中，异常样本可能获得过高的权重，影响最终模型的预测效果。\n",
    "\n",
    "此外，虽然理论上任何学习器都可以作为弱学习器，但实践中最常用的弱学习器是决策树和神经网络。在分类任务中，Adaboost通常使用CART分类树；在回归任务中，则使用CART回归树。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 算法演示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<>:9: SyntaxWarning: invalid escape sequence '\\s'\n",
      "<>:9: SyntaxWarning: invalid escape sequence '\\s'\n",
      "/var/folders/3l/2q08qj3916j6zkcph271llh00000gn/T/ipykernel_2629/291127377.py:9: SyntaxWarning: invalid escape sequence '\\s'\n",
      "  raw_df = pd.read_csv(data_url, sep='\\s+', skiprows=22, header=None)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import AdaBoostRegressor\n",
    "from sklearn import datasets\n",
    "from sklearn import tree\n",
    "import pandas as pd\n",
    "import graphviz\n",
    "\n",
    "data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
    "raw_df = pd.read_csv(data_url, sep='\\s+', skiprows=22, header=None)\n",
    "X  = np.hstack((raw_df.values[::2, :], raw_df.values[1::2, :2]))\n",
    "y = raw_df.values[1::2, 2]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28.04336283 21.9186747  33.12727273 33.12727273 33.12727273 23.8957265\n",
      " 21.00909091 18.53384615 18.23       18.53384615]\n"
     ]
    }
   ],
   "source": [
    "ada = AdaBoostRegressor(n_estimators=3,loss='linear')\n",
    "ada.fit(X,y)# 训练\n",
    "y_ = ada.predict(X) # 预测\n",
    "print(y_[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.2.1 (20241206.2353)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"1366pt\" height=\"373pt\"\n",
       " viewBox=\"0.00 0.00 1366.00 373.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 369)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-369 1362,-369 1362,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<path fill=\"#f7dac4\" stroke=\"black\" d=\"M747.75,-365C747.75,-365 614,-365 614,-365 608,-365 602,-359 602,-353 602,-353 602,-309 602,-309 602,-303 608,-297 614,-297 614,-297 747.75,-297 747.75,-297 753.75,-297 759.75,-303 759.75,-309 759.75,-309 759.75,-353 759.75,-353 759.75,-359 753.75,-365 747.75,-365\"/>\n",
       "<text text-anchor=\"middle\" x=\"680.88\" y=\"-347.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[5] &lt;= 6.92</text>\n",
       "<text text-anchor=\"middle\" x=\"680.88\" y=\"-332.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 89.134</text>\n",
       "<text text-anchor=\"middle\" x=\"680.88\" y=\"-317.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 506</text>\n",
       "<text text-anchor=\"middle\" x=\"680.88\" y=\"-302.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 22.792</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<path fill=\"#fae5d7\" stroke=\"black\" d=\"M575.75,-261C575.75,-261 442,-261 442,-261 436,-261 430,-255 430,-249 430,-249 430,-205 430,-205 430,-199 436,-193 442,-193 442,-193 575.75,-193 575.75,-193 581.75,-193 587.75,-199 587.75,-205 587.75,-205 587.75,-249 587.75,-249 587.75,-255 581.75,-261 575.75,-261\"/>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-243.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[12] &lt;= 14.4</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-228.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 32.051</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-213.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 421</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-198.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 19.573</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M624.56,-296.6C608.67,-287.18 591.24,-276.84 574.84,-267.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"577.01,-264.34 566.62,-262.25 573.44,-270.36 577.01,-264.34\"/>\n",
       "<text text-anchor=\"middle\" x=\"571.51\" y=\"-279.9\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<path fill=\"#eba069\" stroke=\"black\" d=\"M917.75,-261C917.75,-261 784,-261 784,-261 778,-261 772,-255 772,-249 772,-249 772,-205 772,-205 772,-199 778,-193 784,-193 784,-193 917.75,-193 917.75,-193 923.75,-193 929.75,-199 929.75,-205 929.75,-205 929.75,-249 929.75,-249 929.75,-255 923.75,-261 917.75,-261\"/>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-243.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[5] &lt;= 7.445</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-228.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 66.439</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-213.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 85</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-198.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 38.733</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>0&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M736.54,-296.6C752.1,-287.27 769.15,-277.04 785.22,-267.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"787.01,-270.4 793.79,-262.25 783.41,-264.4 787.01,-270.4\"/>\n",
       "<text text-anchor=\"middle\" x=\"789.02\" y=\"-279.93\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<path fill=\"#f7d9c3\" stroke=\"black\" d=\"M317.75,-157C317.75,-157 184,-157 184,-157 178,-157 172,-151 172,-145 172,-145 172,-101 172,-101 172,-95 178,-89 184,-89 184,-89 317.75,-89 317.75,-89 323.75,-89 329.75,-95 329.75,-101 329.75,-101 329.75,-145 329.75,-145 329.75,-151 323.75,-157 317.75,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-139.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[12] &lt;= 7.685</text>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-124.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 16.324</text>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-109.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 240</text>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-94.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 23.007</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M429.71,-194.7C401.48,-183.54 369.49,-170.89 340.44,-159.41\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"341.88,-156.22 331.29,-155.79 339.31,-162.73 341.88,-156.22\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<path fill=\"#fdf6f1\" stroke=\"black\" d=\"M572,-157C572,-157 445.75,-157 445.75,-157 439.75,-157 433.75,-151 433.75,-145 433.75,-145 433.75,-101 433.75,-101 433.75,-95 439.75,-89 445.75,-89 445.75,-89 572,-89 572,-89 578,-89 584,-95 584,-101 584,-101 584,-145 584,-145 584,-151 578,-157 572,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-139.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[7] &lt;= 2.106</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-124.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 16.55</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-109.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 181</text>\n",
       "<text text-anchor=\"middle\" x=\"508.88\" y=\"-94.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 15.021</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M508.88,-192.6C508.88,-185.03 508.88,-176.87 508.88,-168.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"512.38,-168.99 508.88,-158.99 505.38,-168.99 512.38,-168.99\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<path fill=\"#f5cdb0\" stroke=\"black\" d=\"M145.75,-53C145.75,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,0 12,0 12,0 145.75,0 145.75,0 151.75,0 157.75,-6 157.75,-12 157.75,-12 157.75,-41 157.75,-41 157.75,-47 151.75,-53 145.75,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"78.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 17.262</text>\n",
       "<text text-anchor=\"middle\" x=\"78.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 78</text>\n",
       "<text text-anchor=\"middle\" x=\"78.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 26.368</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M190.23,-88.68C172.61,-79 153.42,-68.46 136.02,-58.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"137.78,-55.87 127.33,-54.12 134.41,-62.01 137.78,-55.87\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<path fill=\"#f8dfcc\" stroke=\"black\" d=\"M314,-53C314,-53 187.75,-53 187.75,-53 181.75,-53 175.75,-47 175.75,-41 175.75,-41 175.75,-12 175.75,-12 175.75,-6 181.75,0 187.75,0 187.75,0 314,0 314,0 320,0 326,-6 326,-12 326,-12 326,-41 326,-41 326,-47 320,-53 314,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 7.814</text>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 162</text>\n",
       "<text text-anchor=\"middle\" x=\"250.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 21.388</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M250.88,-88.68C250.88,-80.99 250.88,-72.76 250.88,-64.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"254.38,-64.91 250.88,-54.91 247.38,-64.91 254.38,-64.91\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M489.75,-53C489.75,-53 356,-53 356,-53 350,-53 344,-47 344,-41 344,-41 344,-12 344,-12 344,-6 350,0 356,0 356,0 489.75,0 489.75,0 495.75,0 501.75,-6 501.75,-12 501.75,-12 501.75,-41 501.75,-41 501.75,-47 495.75,-53 489.75,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"422.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 11.411</text>\n",
       "<text text-anchor=\"middle\" x=\"422.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 89</text>\n",
       "<text text-anchor=\"middle\" x=\"422.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 12.547</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M478.55,-88.68C470.65,-79.99 462.11,-70.62 454.16,-61.87\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"456.78,-59.55 447.46,-54.51 451.6,-64.26 456.78,-59.55\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<path fill=\"#fbede3\" stroke=\"black\" d=\"M658,-53C658,-53 531.75,-53 531.75,-53 525.75,-53 519.75,-47 519.75,-41 519.75,-41 519.75,-12 519.75,-12 519.75,-6 525.75,0 531.75,0 531.75,0 658,0 658,0 664,0 670,-6 670,-12 670,-12 670,-41 670,-41 670,-47 664,-53 658,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"594.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 9.873</text>\n",
       "<text text-anchor=\"middle\" x=\"594.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 92</text>\n",
       "<text text-anchor=\"middle\" x=\"594.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 17.414</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M539.2,-88.68C547.1,-79.99 555.64,-70.62 563.59,-61.87\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"566.15,-64.26 570.29,-54.51 560.97,-59.55 566.15,-64.26\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<path fill=\"#efb388\" stroke=\"black\" d=\"M917.75,-157C917.75,-157 784,-157 784,-157 778,-157 772,-151 772,-145 772,-145 772,-101 772,-101 772,-95 778,-89 784,-89 784,-89 917.75,-89 917.75,-89 923.75,-89 929.75,-95 929.75,-101 929.75,-101 929.75,-145 929.75,-145 929.75,-151 923.75,-157 917.75,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-139.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[4] &lt;= 0.659</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-124.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 37.629</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-109.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 48</text>\n",
       "<text text-anchor=\"middle\" x=\"850.88\" y=\"-94.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 33.335</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M850.88,-192.6C850.88,-185.03 850.88,-176.87 850.88,-168.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"854.38,-168.99 850.88,-158.99 847.38,-168.99 854.38,-168.99\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<path fill=\"#e68641\" stroke=\"black\" d=\"M1177.75,-157C1177.75,-157 1044,-157 1044,-157 1038,-157 1032,-151 1032,-145 1032,-145 1032,-101 1032,-101 1032,-95 1038,-89 1044,-89 1044,-89 1177.75,-89 1177.75,-89 1183.75,-89 1189.75,-95 1189.75,-101 1189.75,-101 1189.75,-145 1189.75,-145 1189.75,-151 1183.75,-157 1177.75,-157\"/>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-139.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x[11] &lt;= 395.54</text>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-124.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 16.989</text>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-109.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 37</text>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-94.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 45.735</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>8&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M929.94,-194.98C958.81,-183.66 991.66,-170.77 1021.37,-159.11\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1022.3,-162.51 1030.33,-155.6 1019.74,-155.99 1022.3,-162.51\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<path fill=\"#efaf82\" stroke=\"black\" d=\"M833.75,-53C833.75,-53 700,-53 700,-53 694,-53 688,-47 688,-41 688,-41 688,-12 688,-12 688,-6 694,0 700,0 700,0 833.75,0 833.75,0 839.75,0 845.75,-6 845.75,-12 845.75,-12 845.75,-41 845.75,-41 845.75,-47 839.75,-53 833.75,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"766.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 20.735</text>\n",
       "<text text-anchor=\"middle\" x=\"766.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 45</text>\n",
       "<text text-anchor=\"middle\" x=\"766.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 34.433</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M821.26,-88.68C813.54,-79.99 805.2,-70.62 797.43,-61.87\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"800.16,-59.67 790.9,-54.53 794.92,-64.32 800.16,-59.67\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<path fill=\"#fcefe6\" stroke=\"black\" d=\"M1002,-53C1002,-53 875.75,-53 875.75,-53 869.75,-53 863.75,-47 863.75,-41 863.75,-41 863.75,-12 863.75,-12 863.75,-6 869.75,0 875.75,0 875.75,0 1002,0 1002,0 1008,0 1014,-6 1014,-12 1014,-12 1014,-41 1014,-41 1014,-47 1008,-53 1002,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"938.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 1.742</text>\n",
       "<text text-anchor=\"middle\" x=\"938.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 3</text>\n",
       "<text text-anchor=\"middle\" x=\"938.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 16.867</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>9&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M881.9,-88.68C890.08,-79.9 898.91,-70.42 907.12,-61.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"909.48,-64.2 913.73,-54.5 904.35,-59.43 909.48,-64.2\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M1177.75,-53C1177.75,-53 1044,-53 1044,-53 1038,-53 1032,-47 1032,-41 1032,-41 1032,-12 1032,-12 1032,-6 1038,0 1044,0 1044,0 1177.75,0 1177.75,0 1183.75,0 1189.75,-6 1189.75,-12 1189.75,-12 1189.75,-41 1189.75,-41 1189.75,-47 1183.75,-53 1177.75,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 10.238</text>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 30</text>\n",
       "<text text-anchor=\"middle\" x=\"1110.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 47.157</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1110.88,-88.68C1110.88,-80.99 1110.88,-72.76 1110.88,-64.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1114.38,-64.91 1110.88,-54.91 1107.38,-64.91 1114.38,-64.91\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<path fill=\"#eb9c64\" stroke=\"black\" d=\"M1346,-53C1346,-53 1219.75,-53 1219.75,-53 1213.75,-53 1207.75,-47 1207.75,-41 1207.75,-41 1207.75,-12 1207.75,-12 1207.75,-6 1213.75,0 1219.75,0 1219.75,0 1346,0 1346,0 1352,0 1358,-6 1358,-12 1358,-12 1358,-41 1358,-41 1358,-47 1352,-53 1346,-53\"/>\n",
       "<text text-anchor=\"middle\" x=\"1282.88\" y=\"-35.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">squared_error = 0.148</text>\n",
       "<text text-anchor=\"middle\" x=\"1282.88\" y=\"-20.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 7</text>\n",
       "<text text-anchor=\"middle\" x=\"1282.88\" y=\"-5.7\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 39.643</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>12&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1171.52,-88.68C1189.14,-79 1208.33,-68.46 1225.73,-58.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1227.34,-62.01 1234.42,-54.12 1223.97,-55.87 1227.34,-62.01\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x129400050>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dot_data = tree.export_graphviz(ada[0],filled=True,rounded=True)\n",
    "graph = graphviz.Source(dot_data)\n",
    "graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建第一颗树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第一颗树误差: 0.18233237612196224\n",
      "算法误差: [0.18233238 0.10987582 0.15706842]\n",
      "第一颗树权重: 1.5006246622998372\n",
      "算法权重: [1.50062466 2.09201017 1.68020431]\n"
     ]
    }
   ],
   "source": [
    "# 构建第一课数\n",
    "\n",
    "w1 = np.full(shape=506,fill_value=1/506)\n",
    "y1_ = ada[0].predict(X)\n",
    "\n",
    "# 计算预测值和目标值的误差\n",
    "error_vector = np.abs(y1_ - y)\n",
    "error_max = error_vector.max()\n",
    "if error_max!=0:\n",
    "    error_vector /= error_max # 归一化0～1\n",
    "# 计算算法误差\n",
    "\n",
    "estimator_error = (w1 * error_vector).sum()\n",
    "print('第一颗树误差:',estimator_error)\n",
    "print('算法误差:',ada.estimator_errors_)\n",
    "\n",
    "\n",
    "# 计算算法权重\n",
    "beta = estimator_error / (1. - estimator_error)\n",
    "estimator_weight = np.log(1./beta)\n",
    "print('第一颗树权重:',estimator_weight)\n",
    "print('算法权重:',ada.estimator_weights_)\n",
    "\n",
    "# 根据第一颗树更新权重\n",
    "w2 = w1*np.power(beta,(1.-error_vector))\n",
    "w2 /=w2.sum()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建第二棵树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第二棵树误差: 0.10987581899160484\n",
      " 算法误差: [0.18233238 0.10987582 0.15706842]\n",
      "第二棵树权重: 2.092010172389735\n",
      " 算法权重: [1.50062466 2.09201017 1.68020431]\n"
     ]
    }
   ],
   "source": [
    "y2_ = ada[1].predict(X)\n",
    "\n",
    "error_vector = np.abs(y2_ - y)\n",
    "error_max = error_vector.max()\n",
    "\n",
    "if error_max!=0:\n",
    "    error_vector /= error_max  # 归一化\n",
    "## 计算算法误差\n",
    "estimator_error= (w2*error_vector).sum()\n",
    "print('第二棵树误差:',estimator_error)\n",
    "print(' 算法误差:',ada.estimator_errors_)\n",
    "\n",
    "# 计算算法权重\n",
    "\n",
    "beta = estimator_error / (1 - estimator_error)\n",
    "estimeator_weight = np.log(1./beta)\n",
    "\n",
    "print('第二棵树权重:',estimeator_weight)\n",
    "print(' 算法权重:',ada.estimator_weights_)\n",
    "# 根据第一颗树更新权重\n",
    "w3 = w2*np.power(beta,(1.-error_vector))\n",
    "w3 /=w3.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建第三颗树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第三棵树误差: 0.10987581899160484\n",
      " 算法误差: [0.18233238 0.10987582 0.15706842]\n",
      "第三棵树权重: 2.092010172389735\n",
      " 算法权重: [1.50062466 2.09201017 1.68020431]\n"
     ]
    }
   ],
   "source": [
    "y3_ = ada[1].predict(X)\n",
    "\n",
    "error_vector = np.abs(y3_ - y)\n",
    "error_max = error_vector.max()\n",
    "\n",
    "if error_max!=0:\n",
    "    error_vector /= error_max  # 归一化\n",
    "## 计算算法误差\n",
    "estimator_error= (w2*error_vector).sum()\n",
    "print('第三棵树误差:',estimator_error)\n",
    "print(' 算法误差:',ada.estimator_errors_)\n",
    "\n",
    "# 计算算法权重\n",
    "\n",
    "beta = estimator_error / (1 - estimator_error)\n",
    "estimeator_weight = np.log(1./beta)\n",
    "\n",
    "print('第三棵树权重:',estimeator_weight)\n",
    "print(' 算法权重:',ada.estimator_weights_)\n",
    "# 根据第一颗树更新权重\n",
    "w4 = w3*np.power(beta,(1.-error_vector))\n",
    "w4 /=w4.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预测值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([28.04336283, 21.9186747 , 33.12727273, 33.12727273, 33.12727273,\n",
       "       23.8957265 , 21.00909091, 18.53384615, 18.23      , 18.53384615])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ada.predict(X)[:10]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>y1</th>\n",
       "      <th>y2</th>\n",
       "      <th>y3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28.043363</td>\n",
       "      <td>28.639623</td>\n",
       "      <td>23.895726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21.918675</td>\n",
       "      <td>21.009091</td>\n",
       "      <td>23.895726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28.043363</td>\n",
       "      <td>33.127273</td>\n",
       "      <td>33.221591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28.043363</td>\n",
       "      <td>33.127273</td>\n",
       "      <td>33.221591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28.043363</td>\n",
       "      <td>33.127273</td>\n",
       "      <td>33.221591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>28.043363</td>\n",
       "      <td>21.009091</td>\n",
       "      <td>23.895726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>21.918675</td>\n",
       "      <td>21.009091</td>\n",
       "      <td>18.990355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>18.533846</td>\n",
       "      <td>16.515942</td>\n",
       "      <td>18.990355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>18.533846</td>\n",
       "      <td>16.515942</td>\n",
       "      <td>18.230000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>18.533846</td>\n",
       "      <td>16.515942</td>\n",
       "      <td>18.990355</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          y1         y2         y3\n",
       "0  28.043363  28.639623  23.895726\n",
       "1  21.918675  21.009091  23.895726\n",
       "2  28.043363  33.127273  33.221591\n",
       "3  28.043363  33.127273  33.221591\n",
       "4  28.043363  33.127273  33.221591\n",
       "5  28.043363  21.009091  23.895726\n",
       "6  21.918675  21.009091  18.990355\n",
       "7  18.533846  16.515942  18.990355\n",
       "8  18.533846  16.515942  18.230000\n",
       "9  18.533846  16.515942  18.990355"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y1_ = ada[0].predict(X)[:10]\n",
    "y2_ = ada[1].predict(X)[:10]\n",
    "y3_ = ada[2].predict(X)[:10]\n",
    "display(pd.DataFrame({'y1': y1_, 'y2': y2_, 'y3': y3_}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[28.04336283, 28.63962264, 23.8957265 ],\n",
       "       [21.9186747 , 21.00909091, 23.8957265 ],\n",
       "       [28.04336283, 33.12727273, 33.22159091],\n",
       "       [28.04336283, 33.12727273, 33.22159091],\n",
       "       [28.04336283, 33.12727273, 33.22159091],\n",
       "       [28.04336283, 21.00909091, 23.8957265 ],\n",
       "       [21.9186747 , 21.00909091, 18.99035533],\n",
       "       [18.53384615, 16.51594203, 18.99035533],\n",
       "       [18.53384615, 16.51594203, 18.23      ],\n",
       "       [18.53384615, 16.51594203, 18.99035533]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = np.c_[y1_,y2_,y3_]\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([28.04336283, 21.9186747 , 33.12727273, 33.12727273, 33.12727273,\n",
       "       23.8957265 , 21.00909091, 18.53384615, 18.23      , 18.53384615])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.median(result,axis=1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
